What are the most important questions we need to ask ourselves and how can we use them to learn and inform management? (Gary)
What methods do we need to answer these questions? (Carlo)
What are functions / roles we need about monitoring and evaluation and how should we split them (Sibiry)
What are the most important questions we need to ask ourselves and how can we use them to learn and inform management? (Gary)
Nice overlap between the three different questions. This was quick and I tried to categorize the questions and put a star for unique issues.
Each group came up with high level questions:
What is the contribution of SI to poverty alleviation ?
Did we increase productivity?
Scale: Are our approaches replicable?
Technologies and packages of technology: how do they complement each other? Are there synergies between them?
Who are we helping etc. hh types, landscapes etc.?
Process: Which processes and institutions were helpful? How participatory/top down was it?
Tradeoffs: the farming system, the environment
Adoption - demonstrated which farm is adopted
Partnerships: Can we use existing IPs? Did we improve the efficiciency/effectiveness of different partners?
Learning: What did we researchers learn and what did we learn about farmer learning?
How abut women labor use?
How does intervention complement each other?
Variation and targets (we we are helping, the different domains , household)?
Sustainability ? for household
USAID favorite question , what is worked out and what didn’t work ?
Did Africa rising make contribution in multiple project in the larger context, did it fill the large gap in farming system ?
Collectiveness of information to fill the data collection efficiency ?
Which pathways we provide ?
Sequencing? To talk about farmers adopting in different sequencing, which sequencing is best to achieve impact?
Which advantage do we have on community level?
Unique questions / categories:
What worked and what didn’t work?
Did AfR make a contribution in the context of multiple projects?
Did AfR bridge the gap among commodity projects?
Did we collect enough to complement existing data?
Which pathways did we provide?
Sequencing: farmers adopting in various sequences etc. What is the best sequence to achieve impact?
What are technology advantages for certain groups or others?
Whatto do next aboutthis:
Organizing this to be written down and organized in groups to share with people
Next steps after this is prioritization.
Then we need to pair questions with methods etc and a back-and-forth between implementers and evaluation team.
What methods do we need to answer these questions? (Carlo)
Group 2:
For monitoring, each project has milestones and indicators. The indicators will assess milestones.
For evaluation: qualitative surveys, baseline and follow-up surveys.
We need to discuss the differences between implementing CG centres and IFPRI. We (IFPRI) suggesting controls in other villages to encourage spillovers but that might be different to others. IFPRI might be in charge of collecting data and implementing CG centres too. Efforts need to be coordinated.
Implementers will be administering the project
Group 3:
M&E methods (no distinction between M and E): forms and surveys to be standardized across the 3 megasites, at household level, for input markets, food markets and
3 surveys
Monitor environmental and land use changes, using satellite imagery (hi res)
Management techniques, including soil management.
Central database to collect and capture all data
RCTs ex-post and ex-ante but participation fatigue from farmers so minimize the surveys...
We didn't refer evaluation a lot but we have discussed RCTs, ex-post and ex-ante (why we need ex-ante? you can use it for predicating and screening, with simulation models, it shouldn’t be before the project starts)
We should limit materials
Q: Why are we suggesting ex-ante evaluations? There is no time for ex-ante any more.
A: By ex-ante we're just talking about evaluations going on during the project, not before. It's not about actual measurements.
Group 1:
Monitoring tools - quantitative approaches and strategies (researchers evaluating themselves)
Surveys and discussions with stakeholders, meetings and observations by beneficiaries and researchers
Reports about what needs to be done, what's happening where and how
Keeping track of project activities
Outcome mapping and dynamic monitoring e.g. progress markers
Quantitative methods:
before and after
with or without intervention
RCTs
What are functions / roles we need about monitoring and evaluation and how should we split them amongst us - who should do what? (Sibiry)
Defines standardized indicators and collects procedures
Backstops on sampling design (@MegaSites research planning workshops) and data sharing e.g. technology performance, biophysical data
Evaluation:
IFPRI et al. evaluate the outcomes
They design, execute BLm MT, EL surveys
Report on survey to MS terms (responsible for learning to take place)
Engage national ?? surveys ??
Hires staff to embed in MS terms under IITA / ILRI
Is responsbile for modelling / GIS work towards scaleability
Researchers on the ground are helping but IFPRI are responsible for it all.
Timelines will vary across mega sites. Need to stagger M&E deployment (IFPRI to coordinate).
Possible big gap: Strong coordination between IFPRI and mega site coordination teams. Key scientists from IFPRI should attend the planning workshops and that they hire and embed IFPRI staff in the megasite teams.
Q: What do you mean with IFPRI ensuring data sharing?
A: It's not about IFPRI dispossessing data from scientists but about providing other data...
There is no tension because we work on the same collective goal.
‘What are M&E function and roles we need and how should we split them among us (who should do what)?’ (Sibiry)
The following is a synthesis from the input of 3 successive groups, which organically converged on most issues. The main unknown highlighted by most participants relates to the role of IFPRI in the design and implementation of the M&E system. For simplicity and clarity, results from the discussion hereafter focus on IFPRI’s responsibilities and roles. The responsibilities and roles of other Africa RISING stakeholders (the implementing centers: IITA/ILRI and their collaborators: sister centers, NARS, FOs, NGOs, private sector, etc.) can be deducted. For example, while IFPRI (and its technical collaborators SpatialDev, MSU, ABT, etc.) plays a seminal role in the design, deployment and technical operation of the M&E system, thereafter it is typically the responsibility of mega-site implementers (IITA, ILRI, collaborating centers, NARS, FOs, NGOs, private sector, etc.) to feed the latter with monitoring indicators.
M&E focus
Monitoring
Evaluation
Who
IFPRI (& technical collaborators SpatialDev, MSU, ABT, etc.) backstops on outputs monitoring.
Implementers (IITA, ILRI & al.) are responsible for properly feeding the M&E system with monitoring indicators
IFPRI (& technical collaborators SpatialDev, MSU, ABT, etc.) evaluatesoutcomes and is responsible for evaluation activities. Implementers (IITA, ILRI & al.) assist in the execution thereof.
Funding
IFPRI funds AND delegates with facilitation (co-responsibility)
IFPRI funds AND executes (lead responsibility)
Functions
IFPRI (&al.):
1. Provides efficient technical mechanisms for collation, processing, quality check and serving (mapping) of outputs monitoring indicators
2. Provides a function of ‘independent watchdog’, verifying the indicators reporting by ground research teams
3. Provides a generic set of standardized indicators and indicator collection procedures and tools at various granularity levels (from mega-site to country to district to cluster to village levels)
4. Provides backstopping on research sampling design, data flows and standardization of methods through active participation in mega-site research planning workshops
5. Provides data sharing policy, protocols and vehicles for aggregate (typically: village-level) biophysical and technology performance data
IFPRI (& al.):
1. Is responsible for the design and execution of baseline, mid-term and end-line surveys, including engagement of national statistical offices to help with survey design and administration
2. Is responsible for ensuring interactive learning takes place within and across mega-site teams through regular reporting to the latter
3. Is responsible for organizing modeling and GIS work towards scalability of Africa RISING successful interventions
M&E human resources
IFPRI:
- Hires three M&E officers:
o to be embedded in each of mega-site research teams
o tasked with facilitation of outputs monitoring indicators collection, transfer and serving back to research teams for accelerated learning and efficiency
o tasked with end-to-end organization of baseline, mid-term and end-line surveys, data
o etc. (see TORs provided by IFPRI)
Summary / comments (on day 3) about the results from the presenters
About methods and approaches
We will make use of PMT.
What M&E approaches and methods do we need:
Monitoring
Reports
Dynamic monitoring -keeping track of project activities (what has been done, what has happened, where?)-
Outcome mapping -progress markers-
Forms and surveys need to be standardized across the 3 sites
Surveys at various levels: household, village, input markets, food markets.
Monitor environmental and land-use changes –possibly through the use of high-resolution satellite imagery-
Water and soil management techniques
Central database as data repository
Monitoring is automatic, as each activity is associated to some FtF or custom indicators
Evaluation
Quantitative methods
Instrument: Quantitative surveys (baseline –midpoint –end line) on both treated and control farmers
Methods: Diff-in-Diff and RCTs -taking advantage of local knowledge-. Control farmers should be the same for implementers and IFPRI
Ex-ante and ex-post analysis
Possible trade-off: respondent’s fatigue (AR should minimize the burden of surveys to farmers)
Qualitative methods
Surveys/discussion with stakeholders, both at baseline and follow-up
Meetings and observations
Self-assessment by beneficiaries (agrodealers, researchers)
About roles and responsibilities
We need to look at budgets - so we may not be able to accommodate all wishes.
--> We could be a bit more selective about evaluation but should probably be quite systematic about monitoring.
Africa RISING M&E Expert Meeting
5-7 September 2012Large auditorium, ILRI Ethiopia, Addis Ababa
Back to the event agenda
World cafe discussion on M&E:
The three questions were:What are the most important questions we need to ask ourselves and how can we use them to learn and inform management? (Gary)
Nice overlap between the three different questions. This was quick and I tried to categorize the questions and put a star for unique issues.Unique questions / categories:
What to do next about this:
What methods do we need to answer these questions? (Carlo)
Group 2:- For monitoring, each project has milestones and indicators. The indicators will assess milestones.
- For evaluation: qualitative surveys, baseline and follow-up surveys.
- We need to discuss the differences between implementing CG centres and IFPRI. We (IFPRI) suggesting controls in other villages to encourage spillovers but that might be different to others. IFPRI might be in charge of collecting data and implementing CG centres too. Efforts need to be coordinated.
- Implementers will be administering the project
Group 3:Q: Why are we suggesting ex-ante evaluations? There is no time for ex-ante any more.
A: By ex-ante we're just talking about evaluations going on during the project, not before. It's not about actual measurements.
Group 1:
What are functions / roles we need about monitoring and evaluation and how should we split them amongst us - who should do what? (Sibiry)
Monitoring:
Evaluation:
- IFPRI et al. evaluate the outcomes
- They design, execute BLm MT, EL surveys
- Report on survey to MS terms (responsible for learning to take place)
- Engage national ?? surveys ??
- Hires staff to embed in MS terms under IITA / ILRI
- Is responsbile for modelling / GIS work towards scaleability
Researchers on the ground are helping but IFPRI are responsible for it all.Timelines will vary across mega sites. Need to stagger M&E deployment (IFPRI to coordinate).
Possible big gap: Strong coordination between IFPRI and mega site coordination teams. Key scientists from IFPRI should attend the planning workshops and that they hire and embed IFPRI staff in the megasite teams.
Q: What do you mean with IFPRI ensuring data sharing?
A: It's not about IFPRI dispossessing data from scientists but about providing other data...
There is no tension because we work on the same collective goal.
‘What are M&E function and roles we need and how should we split them among us (who should do what)?’ (Sibiry)
The following is a synthesis from the input of 3 successive groups, which organically converged on most issues. The main unknown highlighted by most participants relates to the role of IFPRI in the design and implementation of the M&E system. For simplicity and clarity, results from the discussion hereafter focus on IFPRI’s responsibilities and roles. The responsibilities and roles of other Africa RISING stakeholders (the implementing centers: IITA/ILRI and their collaborators: sister centers, NARS, FOs, NGOs, private sector, etc.) can be deducted. For example, while IFPRI (and its technical collaborators SpatialDev, MSU, ABT, etc.) plays a seminal role in the design, deployment and technical operation of the M&E system, thereafter it is typically the responsibility of mega-site implementers (IITA, ILRI, collaborating centers, NARS, FOs, NGOs, private sector, etc.) to feed the latter with monitoring indicators.Implementers (IITA, ILRI & al.) are responsible for properly feeding the M&E system with monitoring indicators
1. Provides efficient technical mechanisms for collation, processing, quality check and serving (mapping) of outputs monitoring indicators
2. Provides a function of ‘independent watchdog’, verifying the indicators reporting by ground research teams
3. Provides a generic set of standardized indicators and indicator collection procedures and tools at various granularity levels (from mega-site to country to district to cluster to village levels)
4. Provides backstopping on research sampling design, data flows and standardization of methods through active participation in mega-site research planning workshops
5. Provides data sharing policy, protocols and vehicles for aggregate (typically: village-level) biophysical and technology performance data
1. Is responsible for the design and execution of baseline, mid-term and end-line surveys, including engagement of national statistical offices to help with survey design and administration
2. Is responsible for ensuring interactive learning takes place within and across mega-site teams through regular reporting to the latter
3. Is responsible for organizing modeling and GIS work towards scalability of Africa RISING successful interventions
- Hires three M&E officers:
Summary / comments (on day 3) about the results from the presenters
About methods and approaches
We will make use of PMT.What M&E approaches and methods do we need:
Monitoring
Evaluation
Quantitative methods
- Instrument: Quantitative surveys (baseline –midpoint –end line) on both treated and control farmers
- Methods: Diff-in-Diff and RCTs -taking advantage of local knowledge-. Control farmers should be the same for implementers and IFPRI
- Ex-ante and ex-post analysis
- Possible trade-off: respondent’s fatigue (AR should minimize the burden of surveys to farmers)
Qualitative methodsAbout roles and responsibilities
We need to look at budgets - so we may not be able to accommodate all wishes.--> We could be a bit more selective about evaluation but should probably be quite systematic about monitoring.